Learning Objectives

After completing this lesson, you’ll be able to:

• Carry out a data join with a web-based dataset.
• Write data to a compressed output using the FeatureWriter transformer.
• Upload data to an FTP site (or another online service) using a workspace.

Resources

• CompleteRoads.dwg
  • C:\FMEData\Data\Transportation\CompleteRoads.dwg
• Snowfall.xlsx
  • C:\FMEData\Data\Transportation\Snowfall.xlsx
• Complete workspace
  • C:\FMEData\Workspaces\AdvancedReadingAndWriting\exercise-use-web-based-datasets-complete.fmw

Introduction

In this exercise, we're joining snowfall data to a set of road features and writing the results to an FTP site for other users to access. We'll create a workspace that joins local data to an online dataset, processes the data, and writes it to a different web destination.

1) Add Reader

Start FME Workbench (2025.1 or later), and in a blank workspace, add a reader with these parameters:

The Group Entities parameter is an AutoCAD-specific option. It ensures we have attributes from the AutoCAD source data exposed in Workbench.

The Merged Feature Type option treats all the road data as a single map layer, which is fine because we don't want to handle multiple layers separately.

We are just reading a file from a URL here. If you can access a cloud data storage provider such as Google Drive or Dropbox, try copying CompleteRoads.dwg to a folder on that service. Then, use the Select File From Web functionality to authenticate, connect, and read the data.

2) Locate Snowfall Dataset

In a web browser, browse to the City of Interopolis Open Data Portal (https://s3.amazonaws.com/FMEData/Interopolis/interopolis.html).

Locate the dataset for Snowfall Predictions, right-click on where it says Excel Spreadsheet, and choose to copy the URL:

The exact command will vary by the web browser.

3) Add DatabaseJoiner Transformer

Back in FME Workbench, add a DatabaseJoiner transformer and connect it to the roads feature type. Open the DatabaseJoiner parameters.

Set the parameters as follows:

...where you can paste the URL you copied from the web browser.

Click the Parameters button and check that the data is being read correctly. The Preview table should show records with the correct columns (StreetId, EstimatedSnowfall, etc.).

Back in the main DatabaseJoiner dialog, select Snowfall as the join table and select StreetId as both the Feature Attribute and Table Field to be joined:

If no attributes are available under the Feature Attribute field, you failed to use the Group Entities By: Attribute Schema parameter when adding the AutoCAD reader. To resolve this, the simplest method is to delete and re-add the reader, using the correct options this time.

The final parameters to set in the DatabaseJoiner are Fields to Add and Cardinality:

Select the following Fields to Add:

• EstimatedSnowfall
• LastTreated
• Temperature
• WeatherStation

Ensure Cardinality is set to Match First (1:0...1+); each road feature will be joined to the first matching database record FME finds. This option is least likely to lead to error messages in the log.

The project aims to write the data to KML format. As a training exercise, we're only interested in how and where we write the data. However, a realistic requirement would be to set the color and style of the spatial data being written (in spatial terms, we sometimes call this symbology). Try adding a KMLStyler and setting features' Color using conditional values. For example, make streets with 0 EstimatedSnowfall green, between 0 and 150 yellow, and over 150 red.

4) Add FeatureWriter Transformer

Now, it's time to write the data. Writing data directly to a web service is more complex, so we'll create a zipped, file-based dataset and then upload it to a web service.

So, add a FeatureWriter transformer and connect it to the KMLStyler transformer (if you added one) or the DatabaseJoiner transformer (if you didn't).

Open the parameters and set the writer up as follows:

Then click on the dropdown arrow to the right of the Dataset parameter, and choose the option to Zip Output:

This configuration instructs the transformer to write data directly to a zipped (compressed/archived) file. You can manually type in a path, including .zip if you prefer.

As a final step in this transformer, rename the output feature type to SnowfallPredictions:

When you run the workspace, FME will write a dataset. The FeatureWriter will create an attribute called _dataset to record the dataset's name and path. The next step is to upload this file to a web service.

You can copy the data to any web service you can access; you'll need to use the appropriate connector transformer. For example, use the DropboxConnector to copy data to Dropbox or the GoogleDriveConnector to copy it to Google Drive.

5) Add FTPCaller Transformer

 

Alternatively, you can try using a different Connector transformer with a web service you have access to, like a GoogleDriveConnector or DropboxConnector.

Add an FTPCaller transformer and connect it to the FeatureWriter's Summary output port. The workspace should now look like this (KMLStyler optional):

Open the FTPCaller parameters. For the URL, click on the down-arrow and select Open Text Editor.... 

An overview of the FTPCaller parameters can be found here.

The URL is set to:

ftp://ftp.safe.com/incoming/training/@UUID()/RoadSnowfall.zip

The purpose of the @UUID() is to ensure that you are writing to a unique location each time the workspace is run. This is necessary because the FTP server does not allow overwriting. 

If you want to configure the FTPCaller to dynamically handle the URL based on the value of _dataset from the FeatureWriter, use the following URL:
ftp://ftp.safe.com/incoming/training/@UUID()/@ReplaceString(@ReplaceString(@ReplaceRegularExpression(@Value(_dataset),\.zip(.*),,caseSensitive=TRUE),C:,,caseSensitive=TRUE),\,/,caseSensitive=TRUE).zip

The @ReplaceRegularExpression removes the RoadSnowfall.kml from the _dataset attribute. This is required because the actual path to the file is C:\FMEData\Output\Training\RoadSnowfall.zip, not C:\FMEData\Output\Training\RoadSnowfall.zip\RoadSnowfall.kml.

The rest of the connection string is using two nested @ReplaceString String Functions to convert a file path into a URL path. The inside @ReplaceString replaces the C:\ in the _dataset attribute with a /. This is to ensure that the URL is correct. The outer @ReplaceString replaces all the remaining \ with /.

Finally, we append a .zip to the end of the parameter to replace the .zip removed by the @ReplaceRegularExpression function.

6) Run Translation

Save the workspace and then run it.

In the log file, look for the entry for the FTPCaller.

Above we can see that the FTPCaller uploaded 0.44 MB to the FTP server, and the URL is logged.

Congratulations! You have successfully read and written data using web data connectors.